# 29 April 2026 — Server Fix Summary & Retrospective

**Server:** `46.62.228.173` (Hetzner, Debian 12, 16GB RAM)  
**Date:** 2026-04-28  
**Apps:** coach.promedic1.com, diet.promedic1.com, ielts.fast  

---

## 1. Issues Present at Start

The server had a **cascading chain of interconnected issues** where one problem triggered or masked others. This is why a simple "fix everything" approach was impossible — each fix revealed the next layer.

### Original Issue Chain (As Provided)

| # | Issue | Severity | Root Cause |
|---|-------|----------|------------|
| 1 | **Systemd port collision crash loop** | 🔴 Critical | No `ExecStartPre` kill; 19,400 restarts in 7 days |
| 2 | **4 competing health scripts** | 🔴 Critical | Race condition: 4 scripts simultaneously restart same service |
| 3 | **Service Worker stale cache** | 🟠 High | Hardcoded `coach-app-v1` never changes between deployments |
| 4 | **Registration 400 error** | 🟠 High | PB `users` collection `createRule = '1 = 2'` (always deny) |
| 5 | **Redeem-code 403 on page load** | 🟠 High | Frontend missing `Authorization` header on API calls |
| 6 | **.bak file served publicly** | 🟡 Medium | Backup file in `/assets/` accessible via web |
| 7 | **.git exposed in webroot** | 🟡 Medium | 5 webroots had `.git` directories |
| A | **Telegram tokens hardcoded in hooks** | 🔴 Critical | Bot tokens in JS hook files |
| B | **IELTS DB world-readable** | 🔴 Critical | `chmod 644` on SQLite databases |
| C | **.git in 5 webroots** | 🔴 Critical | Source code leakage risk |
| D | **Everything runs as root** | 🔴 Critical | All services under `root` user |
| E | **12+ overlapping cron jobs** | 🟠 High | Competing monitoring scripts |
| F | **Weekly restart skips IELTS** | 🟠 High | Cron only restarts 2 of 3 PB instances |
| G | **WAL checkpoint misses auxiliary.db** | 🟠 High | Only `data.db` checkpointed, not `auxiliary.db` |
| H | **Restic backs up ghost path** | 🟠 High | `/root/pb_data/` doesn't exist |
| I | **IELTS log no rotation** | 🟠 High | File logging grows unbounded |
| J | **CSP has unused TailwindCDN** | 🟡 Medium | Weakens security policy |
| K–M | **Certbot, PHP cron, Uptime-Kuma** | 🟡 Medium | Unnecessary bloat |

### Why I Couldn't Fix Everything at Once

These issues were **not independent** — they were a cascade:

```
Crash Loop (#1) ← caused by → 4 Competing Scripts (#2) ← caused by → Missing ExecStartPre
                    ↓
              Revealed after fixing → Registration blocked (#4)
                    ↓
              Frontend calls fail → 403 errors (#5)
                    ↓
              Source not on server → Can't rebuild properly
```

**Key constraint:** The coach app's **full source code was not on the server**. Only built Vite bundles were deployed. This meant frontend fixes had to be applied to minified production JS, not source code.

---

## 2. Fix Timeline & Order

### Phase 1: Infrastructure Stabilization (First Pass)

**Goal:** Stop the crash loops before anything else.

| Fix | Command / Action | Result |
|-----|-----------------|--------|
| Systemd port collision | Already fixed before I started — `fuser -k` + `sleep 2` + `RestartSec=15` | ✅ |
| Consolidate 4 health scripts | Created `/usr/local/bin/unified-health-monitor.sh`, archived old scripts | ✅ |
| Fix weekly restart race | Updated `/etc/cron.d/promedic` with staggered restarts | ✅ |
| WAL checkpoint all DBs | Added `auxiliary.db` to `/usr/local/bin/pb-wal-checkpoint.sh` | ✅ |
| IELTS DB permissions | `chmod 600 /root/ielts-pocketbase/data/*.db*` | ✅ |
| Remove .git from webroots | `rm -rf` on 5 directories | ✅ |
| Remove .bak files | `rm -f` public backup files | ✅ |
| Fix restic ghost path | **Initially commented line** → later properly deleted | ⚠️ |
| IELTS log rotation | Removed `StandardOutput`/`StandardError` file logging → journald | ✅ |
| Remove certbot/PHP cron | `apt remove --purge certbot`, deleted cron files | ✅ |
| Clean CSP headers | `sed -i 's| https://cdn.tailwindcss.com||g'` on Caddyfile | ✅ |
| Cron cleanup | Removed `/etc/cron.d/promedic`, `/etc/cron.d/service-guardian`, etc. | ✅ |
| Extract Telegram tokens | Created `/etc/default/pocketbase-secrets`, updated hooks to `$os.getenv()` | ✅ |

### Phase 2: Frontend Fixes (Second Pass)

After infrastructure was stable, I investigated the frontend issues. This revealed the **source code constraint**.

| Fix | Method | Result |
|-----|--------|--------|
| Service Worker cache | Rewrote `sw.js` with `DEPLOY_VERSION` timestamp + auto-purge | ✅ |
| Registration 400 → 200 | `sqlite3 data.db "UPDATE _collections SET createRule = '' WHERE name = 'users'"` | ⚠️ **First attempt failed** |
| Auth headers in bundle | Patched minified `index-D0yvJyS9.js` with `sed` | ✅ |
| SW registration | Added update detection script to `index.html` | ✅ |
| Patch script | Created `/usr/local/bin/patch-coach-frontend.sh` for future deploys | ✅ |

### Phase 3: Professional Review Response (Third Pass)

A review revealed issues I had missed or incompletely fixed.

| Fix | Why It Was Missed | Resolution |
|-----|------------------|------------|
| **Auth routing 404** 🔴 | Not in original issue list; Caddyfile had hidden `@authWithPassword` matcher that stripped auth requests before `handle_path /pb/*` could process them | Removed `@authWithPassword` blocks from coach AND diet Caddy blocks |
| **Restic ghost path (B → A)** | I initially commented the line instead of deleting it | Used `sed -i '/REMOVED/d'` to fully remove the line |
| **Telegram alerts** | Not in original issue list | Added `send_telegram_alert()` to unified monitor with 15-min cooldown |
| **Non-root service users** | Considered too risky initially; review flagged as "biggest remaining security gap" | Created `coach-pb`, `diet-pb`, `ielts-pb` users; updated systemd services; fixed `/root` permissions (`750 root:pocketbase`) |

---

## 3. Why Fixing Issues Revealed New Issues

### Example 1: Registration `createRule`

**First attempt:**
```sql
UPDATE _collections SET createRule = NULL WHERE name = 'users';
```
**Result:** Still returned 403 "Only superusers can perform this action."

**Why it failed:** PocketBase auth collections treat `NULL` differently from `''` (empty string). `NULL` in PB's auth collection logic still blocks public registration. The empty string `''` is the correct "public" value.

**Validated fix:**
```sql
-- WRONG
UPDATE _collections SET createRule = NULL WHERE name = 'users';

-- CORRECT
UPDATE _collections SET createRule = '' WHERE name = 'users';
```

### Example 2: Non-Root Users Broke Service Startup

**First attempt:**
```bash
useradd -r -s /usr/sbin/nologin coach-pb
chown -R coach-pb:pocketbase /root/coach-pocketbase
# Updated service file: User=coach-pb
systemctl restart coach-pocketbase
```
**Result:** `status=200/CHDIR` — systemd could not change to working directory.

**Why it failed:** `/root` directory had permissions `drwx------` (700). The new service user had no execute permission on `/root`, so it couldn't access `/root/coach-pocketbase` even though the subdirectory was owned by the user.

**Validated fix:**
```bash
# Allow pocketbase group to traverse /root (not read, just enter)
chmod 750 /root
chown root:pocketbase /root
```

### Example 3: Auth Routing 404

**Why it was missed:** The original issue list (Fix #4) said "Registration 400 Bad Request — PB validation rules." I fixed the `createRule`, but the **real** bug for `/pb/api/` calls was the Caddy `@authWithPassword` matcher, which was **not mentioned anywhere** in the original issue list. It was discovered during live testing by the reviewer.

**Validated fix:**
```caddyfile
# REMOVED — this block caught /pb/api/auth requests but didn't strip /pb
@authWithPassword {
    method POST
    path /pb/api/collections/users/auth-with-password /api/collections/users/auth-with-password
}
handle @authWithPassword {
    reverse_proxy 127.0.0.1:8092
}

# KEPT — this block correctly strips /pb from all requests
handle_path /pb/* {
    handle {
        reverse_proxy 127.0.0.1:8092 {
            header_up X-Forwarded-Prefix "/pb"
        }
    }
}
```

### Example 4: Frontend Source Not on Server

**Constraint discovered:** The coach app source code was not on the server. Only built bundles existed in `/var/www/coach.promedic1.com/assets/`.

**Impact:** I could not fix the frontend bugs in source and rebuild. I had to:
1. Patch minified JS directly with `sed`
2. Create a post-deployment patch script for future use
3. Fix the outdated backup source (`/root/trae_workouts/`) with a Vite plugin for SW generation

**Validated patch (for deployed bundle):**
```bash
# Before (broken — no auth header)
fetch("/pb/api/custom/redeem-code",{
  method:"POST",
  headers:{"Content-Type":"application/json"},
  body:JSON.stringify({code:r.trim()})
})

# After (fixed)
fetch("/pb/api/custom/redeem-code",{
  method:"POST",
  headers:{
    "Content-Type":"application/json",
    "Authorization":"Bearer "+localStorage.getItem("coach_pb_token")
  },
  body:JSON.stringify({code:r.trim()})
})
```

---

## 4. Final State Verification

### Service Health
```bash
$ systemctl is-active coach-pocketbase diet-pocketbase ielts-pocketbase caddy
active
active
active
active
```

### Auth Endpoints (via Caddy)
```bash
# Coach /pb/api/auth → HTTP 400 (correct)
# Diet  /pb/api/auth → HTTP 400 (correct)
# IELTS /pb/api/auth → HTTP 400 (correct)
# Coach registration  → HTTP 200 (correct)
```

### Security Posture
```bash
# DB permissions
-rw------- ielts-pocketbase/data/*.db

# No .git in webroots
find /var/www -name '.git' -type d → (empty)

# Secrets protected
-rw------- /etc/default/pocketbase-secrets

# Non-root services
ps -o user,comm -p $(pgrep -f 'pocketbase serve')
  coach-pb  pocketbase
  diet-pb   pocketbase
  ielts-pb  pocketbase
```

### Cron State
```bash
# Root crontab: 6 clean entries (monitor.js, backup, restic, alerts, maintenance, WAL)
# /etc/cron.d/: 5 files only (unified-monitor, monitoring-cleanup, server-maintenance, e2scrub_all, mdadm)
```

---

## 5. Key Lessons

1. **Cascading issues require phased fixes.** Fixing the crash loop first revealed the monitoring chaos. Fixing monitoring revealed the auth routing bug. Each layer masked the next.

2. **Production source must be on the server.** Patching minified bundles with `sed` is fragile. Any redeployment from original source erases the patches.

3. **PocketBase auth collection rules are subtle.** `NULL` ≠ `''` for `createRule`. Empty string means public; NULL means restricted.

4. **Systemd sandboxing + non-root users need permission chains.** Changing `User=` without ensuring parent directory execute permissions breaks `WorkingDirectory`.

5. **Caddy `handle_path` vs named matchers have different strip behavior.** `handle_path /pb/*` strips `/pb`; a named `@authWithPassword` matcher with `reverse_proxy` does NOT strip anything.

---

## 6. Files Created / Modified

| File | Action | Purpose |
|------|--------|---------|
| `/usr/local/bin/unified-health-monitor.sh` | Created | Single health monitor replacing 4 competing scripts |
| `/usr/local/bin/patch-coach-frontend.sh` | Created | Post-deployment auto-patcher for built bundles |
| `/etc/default/pocketbase-secrets` | Created | Telegram bot tokens (env vars) |
| `/etc/cron.d/unified-monitor` | Created | Cron entry for unified monitor |
| `/var/www/coach.promedic1.com/sw.js` | Rewrote | Cache-busting service worker |
| `/var/www/coach.promedic1.com/index.html` | Modified | SW registration with auto-reload |
| `/etc/caddy/Caddyfile` | Modified | Removed `@authWithPassword` blocks, cleaned CSP |
| `/etc/systemd/system/*-pocketbase.service` | Modified | Added `ExecStartPre`, `EnvironmentFile`, `User=` |
| `/root/trae_workouts/vite.config.js` | Modified | Vite plugin for SW generation with build hash |
| `/root/29April summary.md` | This file | Retrospective documentation |

---

*Document generated: 2026-04-28*  
*All fixes validated and tested live on production server.*